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About the research 

Mapping adult literacy performance 

Michelle Circelli, David D Curtis, National Centre for Vocational Education Research, 
and Kate Perkins, Kulu Adventures in Management 

Both national and international research demonstrate the relationship between increasing levels of 
language, literacy and numeracy proficiency and positive outcomes, such as improved economic 
performance and social cohesion. Being able to measure the level of proficiency in these skills and any 
changes in their levels is important for getting a sense of how well language, literacy and numeracy 
programs are working. 

In Australia there are a number of tools used to measure language, literacy and numeracy proficiency. 
Among these are the Adult Literacy and Life Skills (ALLS) survey conducted by the Australian Bureau of 
Statistics (ABS) and the Australian Core Skills Framework (ACSF). 

The primary purpose of the Adult Literacy and Life Skills survey is to identify and measure the literacy 
and numeracy skills of adult populations within and across a number of participating Organisation for 
Economic Co-operation and Development (OECD) countries. However, it has two drawbacks. First, it is 
relatively coarse and is designed to provide a summary of literacy and numeracy rather than act as an 
assessment tool. Second, it is only administered every ten years. 

By contrast, the Australian Core Skills Framework was designed to be used as an assessment tool at the 
student level. It makes learning relevant to the individual and provides evidence of progress, so that a 
learner's performance in a core skill can be assessed and their strengths and weaknesses identified. 

The framework is used in two key federal government programs — the Language, Literacy and 
Numeracy Program and the Workplace English Language and Literacy program — and in a variety of 
other settings, including the South Australian Certificate of Education. 

This paper outlines a study the National Centre for Vocational Education Research (NCVER) is 
undertaking to determine whether performance levels on the literacy and numeracy scales of the Adult 
Literacy and Life Skills survey can be reliably mapped to the performance levels of the Australian Core 
Skills Framework. The initial stage is promising, with an expert panel concluding that it is feasible to 
map Adult Literacy and Life Skills survey items to Australian Core Skills Framework levels. The second 
stage will involve a larger-scale study to empirically align the two frameworks. If this stage is 
successful, then various possibilities arise for monitoring the language, literacy and numeracy skills of 
various groups against the Adult Literacy and Life Skills survey scale. 



Tom Karmel 

Managing Director, NCVER 
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Introduction 



The National Centre for Vocational Education Research (NCVER) is undertaking a study to determine 
whether it is possible to align the performance levels of two frameworks for measuring the language, 
literacy and numeracy skills of adults; namely, the Adult Literacy and Life Skills (ALLS) survey and the 
Australian Core Skills Framework (ACSF). Both frameworks have five levels of performance and it is 
sometimes assumed that these levels are egual. But are they? 

Measures of adult literacy 

The Adult Literacy and Life Skills survey, and its predecessor, the International Adult Literacy Survey 
(IALS) were developed to enable the collection of comparable international data on literacy and 
numeracy proficiency. In 1992 the Organisation for Economic Co-operation and Development (OECD) 
stated that, internationally, low literacy levels were having a significant impact on economic 
performance and social cohesion at an international level. But trying to get a better sense of the 
extent of literacy problems, and the policy implications that would arise from these, was thwarted by 
the lack of data (cited in National Center for Education Statistics 1998, p. 13). 

The focus of the International Adult Literacy Survey, the Adult Literacy and Life Skills survey, and the 
current survey, the Programme for the International Assessment of Adult Competencies (PIAAC), is 
always on the skills that an individual needs to participate fully and successfully in a modem society. 
These types of surveys are designed to provide performance information at aggregate levels such as the 
adult population and by important sub-groups (for example, gender, location). Further, given the cost 
associated with the management and administration of such large-scale international surveys, the 
period of time between surveys is generally long, in the order of five to ten years. In Australia, the 
International Adult Literacy Survey was administered in 1996, the Adult Literacy and Life Skills survey 
in 2006 and the Programme for the International Assessment of Adult Competencies is scheduled for 
late 2011. While such surveys provide important information about Australia's skills position relative to 
other countries, this timeframe does not permit the close monitoring of progress against national goals. 

The Australian Core Skills Framework (ACSF), released in December 2008, describes performance in the 
five core skills of reading, writing, oral communication, numeracy and learning. 1 It is intended to act as 
a national framework for describing and discussing English language, literacy and numeracy 
performance, with benchmarks against which to assess and report on the progress of individuals or 
learner cohorts. The framework is built on the National Reporting System for Adult Literacy and 
Numeracy (NRS) — a tool used since 1995 for reporting outcomes from adult English language, literacy 
and numeracy (LLN) provision in the federally funded Language, Literacy and Numeracy Program (LLNP) 
and the Workplace English Language and Literacy (WELL) program. Although the Core Skills Framework 
is much broader in scope than the National Reporting System, it maintains the five performance levels 
developed for the National Reporting System. 

While the use of the Australian Core Skills Framework was not originally mandated in situations other 
than the Language, Literacy and Numeracy and the Workplace English Language and Literacy programs. 



1 The focus on the ACSF does not downplay the importance of alternative adult literacy and numeracy frameworks such as 
tests of vocational English proficiency. 
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it is being adopted across a range of contexts and for a range of purposes. All general education 
curricula have been mapped to the ASCF; South Australia has adopted the framework as part of the 
South Australian Certificate of Education (SACE), using attainment of exit level 3 in each core skill as a 
minimum reguirement for achieving the certificate; and Victoria University has adopted the framework 
as part of its whole- of-university strategy to support students' literacy and numeracy skills 
development. In the vocational education and training (VET) sector, several industry skills councils are 
sponsoring national professional development on the ACSF for trainers in their fields, and the 
Department of Education, Employment and Workplace Relations (DEEWR) has recently funded industry 
skills councils to map training package units to the framework. 

A main difference between the ALLS and the ACSF is the assessment purpose. The large-scale ALLS 
survey is a summative and evaluative tool. That is, it is used to give a summary of learners' knowledge 
and skill at a point in time and does not provide feedback to inform future learning. 

The Core Skills Framework can be used as either a summative or a formative tool. At any point in time, 
a learner's performance in a core skill can be measured against the descriptors (called 'indicators' and 
'performance features') associated with each of the five levels, and a level of performance assigned. In 
terms of its use as a formative or diagnostic tool, any activity or test can become an assessment 
instrument if it is mapped to the ACSF and then used to identify an individual' s specific strengths and 
weaknesses. The performance features offer a means of providing detailed performance feedback and 
identifying where the focus of subseguent effort might yield useful results. Progress over time can be 
monitored against the levels, and also against specific indicators and performance features. 

Although the ACSF has been used primarily as a formative tool, there is increasing interest in its 
summative capacity. For example, 

■ Kangan- Batman TAPE has recently developed a series of contextualised assessment instruments 
described in ACSF terms, which are being trialled as tools to identify commencing students' 
language, literacy and numeracy performance. 

■ Victoria University is trialling ACSF-based contextualised activities to establish language, literacy 
and numeracy performance benchmarks for commencing students as a precursor to tracking, 
monitoring and measuring performance improvement over time. 

■ The Australian Council for Educational Research has benchmarked its 'vocational indicator' 
assessment instrument to the ACSF. 

Why is this project being undertaken? 

In late 2008, as part of the National Skills and Workforce Development Agreement, a Council of 
Australian Governments (COAG) directive specified that the proportion of the working- age population 
with low foundation skill levels be reduced to enable effective educational, labour market and social 
participation, and that the proportions at ALLS levels 1, 2 and 3 be monitored as a means of checking 
progress (Council of Australian Governments 2008). That is, the objectives are stated in terms of ALLS 
survey levels. 

However, as discussed earlier, key federal government programs such as the Language, Literacy and 
Numeracy and the Workplace English Language and Literacy programs routinely use the ACSF to assess 
the state and progress of individual or group literacy and numeracy skills. It must be noted that these 
programs provide information on a very small proportion of the population — some 100 000 per year — 
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who fall within the target area specified by the Council of Australian Governments. Of other initiatives 
with a language, literacy and numeracy focus that could be used to provide further information on the 
literacy and numeracy progress of various learner groups, none can be measured and reported on using 
the ALLS or PIAAC tests but they could be monitored against the ACSF benchmarks. 

Thus, this project aims to investigate whether performance levels on the ALLS literacy and numeracy 
scales can be reliably mapped to the performance levels of the ACSF, essentially meaning that ACSF 
performance levels could be used as a proxy for ALLS performance levels. 2 This would make it possible 
to provide more freguent information on the literacy and numeracy development of identified target 
groups of the adult population than is available from the large-scale international testing programs. 

Has such a mapping exercise been done before? 

In 2002 an exercise to map the National Reporting Service against the International Adult Literacy 
Survey was undertaken, since it seemed, on the surface at least, that the two measures were directly 
comparable, as both had five levels of competence. A general alignment was found but not a direct 
one-to-one relationship between the five levels of each scale. Indeed, there were no IALS tasks at NRS 
level 1 (Hagston 2002). 

Literacy assessment in the two frameworks 3 

The ALLS scale is constructed around a few key aspects that contribute to the difficulty in reading and 
responding to text. These include: 

■ the length and density of the text 

■ the number of pieces of information that are reguired; the number of pieces of information that are 
presented (that is, distractors) 

■ the extent to which the reader must generate inferences based on given information or use assumed 
knowledge 

■ the presence of pointers, such as headings (ABS 2008, p. 77). 

All these aspects come together to form a single cluster of factors that contribute to item difficulty, so 
that for each item a single score of known precision is reported. 

The ACSF has four principles that are taken into account when determining performance. These are: 

■ the nature and degree of support provided 4 

■ prior knowledge and familiarity of context, text, task and topic 



2 It is important to note that the purpose of this project is not to evaluate the utility of either the ACSF or ALLS but rather 
to evaluate their possible commensurability. 

3 Further information about the ALLS is provided in appendix A. Additional information about aspects of performance, 
assessment and feedback in the ACSF is provided in appendix A. For a copy of the ACSF please go to DEEWR Australian 
Core Skills Framework (<http:// www.deewr.gov.au/ Skills/ Programs/ LitandNum/ Workplace EnglishLanguageandLiteracy/ 
Documents/ Australian Core SkillsFramework. pdf >) 

4 When the ACSF is being used for summative purposes, it is assumed that the learner can perform to a particular level 
without support. If the ACSF is being used for formative purposes and some support is provided, then this is taken into 
account when determining the learner's performance level. 
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■ the complexity of the text, including its length, specificity, information content, vocabulary and 
degrees of abstraction 

■ the task complexity, including the number of pieces of information reguired, number of processes 
involved and the extent of reguired inference (Department of Education, Employment and 
Workplace Relations 2008, p. 8). 

It would appear then that there are substantial similarities between the factors that contribute to task 
difficulty in both the ALLS survey and the ACSF. Both frameworks include the length and complexity of 
the text, its information content (density) and the extent to which the reader is reguired to generate 
inferences. This compatibility suggests that at least gualitative mapping of the performance levels 
between the two frameworks is feasible. 

About the project 

There are two stages to this project. The first was a feasibility study undertaken in 2010 to determine 
whether a potential mapping between the two frameworks is reasonable. This involved an expert 
group, including developers of the ACSF, together with an experienced item developer and a literacy 
practitioner. The group was asked to assess where a number of prose and document literacy and 
numeracy ALLS items fitted within the ACSF framework, 5 based on the assumption that a learner would 
attempt to perform the tasks independently. Since ALLS items have known locations on its relevant 
scale, the consensus j udgment of panel members provided a gualitative link between the two scales. 6 

For this phase, items that represented levels 1 and 2 and the lower part of level 3 of the ALLS prose 
and document literacy and numeracy scales were used, since individuals whose literacy performance 
lies within this range have tended to be of most interest in programs that use the ACSF as a tool in 
literacy improvement. 7 

It was found that there was only one instance of a member of the expert panel having difficulty in 
assigning an ACSF level to an ALLS prose, document or numeracy item. Therefore, in general, it is 
reasonable to assume that the performance levels of ALLS items can be mapped to the ACSF levels. 

The second stage of the project involves a larger-scale research study to empirically align the two 
frameworks onto a single scale for the domains of reading and numeracy, in an online survey, 
teachers/ tutors familiar with adult literacy and numeracy concepts will anonymously rate a student 
whose literacy and/ or numeracy levels are most familiar to them against statements and sample tasks 
drawn directly from both the ACSF and ALLS frameworks. The ratings can then be analysed and, using 



5 For the feasibility phase a version of the Delphi method (linstone & Turoff 2002) was used. Further information about 
the Delphi method is provided in appendix B, along with qualitative descriptions of performance at the various levels of 
the two frameworks and a brief description of the ALLS items used in this study. 

6 A quantitative link could be generated, but this would require the judgments of many experts. In the first phase of the 
research, the focus is on establishing whether qualitative mapping is feasible before possibly committing to the 
complexity and cost of a quantitative mapping activity. 

7 The items used were developed by Statistics Canada and the Educational Testing Service (ETS). Items used in IALS, ALLS 
and PIAAC are confidential as it is important to be able to use common items across assessment occasions in order to 
monitor progress in literacy achievement over time. Statistics Canada and the ETS allowed NCVERto use the items, 
subject to confidentiality undertakings being entered into by all project participants. The items used in this study were 
selected by Statistics Canada and the Educational Testing Service and are not required for the PIAAC. NCVERis grateful 
to Statistics Canada and the ETS for permission to use their items and for their assistance in selecting items for the 
current study. 
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Item Response Theory, placed onto the same scale of measurement. 8 This is the most direct method for 
determining and comparing the complexity of the two frameworks. 

And then what? 

We do not yet know what the outcomes of this second stage will be, but it would not be unreasonable 
to assume that a relatively close alignment between the performance levels of the two frameworks will 
occur. If this is the case, then ACSF-based information from various interventions could be reported, if 
reguired, against the ALLS-based targets specified by the Council of Australian Governments. Some of 
the options are discussed below. 

Monitoring the progress of those with very low language, literacy and numeracy 
skills 

As indicated earlier, at the time of writing only two federally funded adult literacy programs are 
reguired to report against the ACSF: the Language Literacy and Numeracy and the Workplace English 
Language and Literacy programs. 

■ Potential LLNP participants are assessed against ACSF benchmarks and enter different streams of 
the program depending on their ACSF level. The maj ority are at ACSF pre-level 1, level 1 or level 2 
in most or all core skills, but some entering the vocational stream may already be at ACSF level 3 in 
some core skills. Providers must show evidence that a learner has progressed over the initial 200 
hours of language, literacy and numeracy training. For those starting at very low levels, this is 
measured by an increase in at least two indicators within two different core skills. Although 
theoretically it should be possible to aggregate these data to provide a snapshot of the progress of 
the 100 000 participants a year, it is unclear whether the current data-capture software has the 
capability to deliver this kind of information. However, providers do give these data as part of their 
reports, so it could be collated. 

■ In the WELL program, there is no baseline assessment of either individuals or groups, but the ACSF is 
used to describe the performance of a WELL cohort at the end of the training program. If expressed 
in ALLS terms, the exit scores could provide some comparable information on the language, literacy 
and numeracy performance of groups of workers. However, the data would only be useful for 
measuring the impact of the program itself if entry-level benchmarks were also established. 

■ The implementation of a national Foundation Skills Training Package may make it possible to gather 
detailed and consistent information about the language, literacy and numeracy progress and 
performance of specific target groups, many of whom may be members of, or aspiring to enter, 
specific industries. 

■ All students sitting for the South Australian Certificate of Education are reguired to demonstrate 
literacy and numeracy skills to a minimum of ACSF exit level 3, either through explicitly designed 
literacy and numeracy units mapped to the ACSF or through SACE- endorsed literacy- and numeracy- 
rich subj ects, in which C grade is eguated with ACSF exit level 3. A study of the success rates of this 



Item Response Theory (IRT) encompasses the design, analysis and scoring of tests, questionnaires and similar instruments 
for measuring abilities, attitudes, or scales that describe a particular concept, in this case, adult literacy and numeracy. 
It is based on the application of related mathematical models to testing data. Item Response Theory underpinned the 
development of the ALLS scale and is also is the preferred method for the development of tests such as the National 
Assessment Project — Literacy and Numeracy (NAPLAN) and the OECD Programme for International Student Assessment 
(PISA). 
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large cohort of senior secondary students in South Australia and the Northern Territory could 
provide insight into the potential of this approach to raise the literacy and numeracy skills of young 
adults in the school sector. Given the evidence that the SACE reguirements are already having an 
impact on approaches adopted in middle schooling, there is potential for the longitudinal tracking 
of the impact of a formal language, literacy and numeracy policy with set minimum reguirements in 
the secondary schooling sector. 

Monitoring the LLN skills of those in vocational education and training 

Learner cohorts in vocational education and training are operating across all ACSF skill levels. 
Streamlined training packages are to have an explicit focus on 'foundation skills', although at the time 
of writing an agreed definition of this term is still under discussion. While it is generally understood 
that it will involve an explicit focus on language, literacy and numeracy, it has not yet been decided if 
this will be expressed in ACSF terms. However, the Department of Education, Employment and 
Workplace Relations has made funding available to industry skills councils to map training packages to 
the ACSF, so the data are likely to be available, even if the ACSF itself does not become an officially 
endorsed component. 

A pilot study has shown the potential of the mapping process to provide ACSF-based information that 
could be directly relevant to the Council of Australian Governments targets if expressed in ALLS terms. 
For example, it can now be demonstrated that, in order to complete a Certificate III in 
Electrotechnology, a learner will have developed and demonstrated skills in numeracy to ACSF level 5 
and reading, oral communication and writing to ACSF level 4. Thus, anyone achieving this gualification 
can be assumed to be operating above ALLS levels 3 in reading and numeracy — at least in the 
language, literacy and numeracy relevant to their disciplines. If all training packages are mapped 
consistently to the ACSF, completions data could be used to show the minimum language, literacy and 
numeracy performance in ACSF terms of anyone completing training units in a particular industry 
sector. A closer mapping of the ALLS survey and the ACSF would allow this information to be translated 
into ALLS terms. 

The ACSF mapping of training packages is only one part of a broader picture. The Core Skills Framework 
is increasingly being used as the basis of language, literacy and numeracy training for VET practitioners. 
There is also a growing emphasis on the pre- and post-entry LLN assessment of learners undertaking 
vocational training, using generic and contextualised assessment tools benchmarked against the ACSF. 
Again, this provides an opportunity to conduct longitudinal research in identified target areas to 
provide evidence of the impact of various interventions. Such research might use ACSF benchmarks to 
assist trainers and learners to identify specific areas of strength and weakness and thus inform the 
design of teaching methodologies, as well as to identify ALLS reference points to provide a snapshot of 
the language, literacy and numeracy progress of those engaged in formal training in the industry in 
guestion. 

Monitoring language, literacy and numeracy progress at undergraduate levels 

There is a growing concern about the LLN skills of students entering undergraduate courses, with over 
one-third of universities conducting or planning to introduce post-entry language, literacy and 
numeracy assessment (Dunworth 2009). The focus on language, literacy and numeracy is about to 
increase significantly following the announcement by the Australian Universities Quality Agency (now 
the Tertiary Education Quality and Standards Agency) that it will be evaluating language, literacy and 
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numeracy processes and outcomes, and will be looking for evidence that students have improved 
discipline-related language, literacy and numeracy skills during their course of study. 

Victoria University is the first university to adopt the ACSF as the platform for a whole-of-university 
language, literacy and numeracy strategy. At the time of writing, the aim is to use the framework to: 

■ map the context- specific LLN reguirements of courses in both vocational and higher education 

■ inform the design of course materials, methodologies and assessment to support students' LLN skill 
development in discipline and career- relevant areas 

■ benchmark the post-entry LLN performance of all students, with a particular emphasis on the 
aspects most relevant to their chosen course of study and career 

■ monitor student progress over time. 

While it is not yet clear whether other universities will follow suit, Victoria University could provide 
one site for the evaluation of the impact of a systematic approach to LLN across both vocational and 
higher education. While post-entry and exit data will be expressed in ACSF terms, it could also be 
translated into ALLS performance levels if the mapping proposed in this study is successful. 

Data capture 

While we can see there are potentially many sources of language, literacy and numeracy data, this 
leads to further guestions about how we might best capture ACSF data to use as evidence of literacy 
and numeracy progress against the ALLS survey and its successors. It may, for example, reguire the 
development of a new data- collection system, perhaps as part of a research program incorporating case 
studies and broader statistical sampling. 
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Conclusion 



The ACSF is increasingly being used in a variety of settings and has the potential to become the bridge 
between international measures such as the ALLS survey and its successors and the reality of language, 
literacy and numeracy performance in identified Australian contexts. 

However, it should also be noted that neither ALLS nor ACSF data alone will change language, literacy 
and numeracy skills performance. Providing adults in different walks of life with the opportunity to 
achieve a prescribed benchmark of adult literacy and numeracy performance may reguire a greater 
investment in support per individual than is currently being committed and/ or changes to funding 
structures. It will need a larger and possibly more diverse adult literacy workforce than exists now and 
a broadening of the contexts within which language, literacy and numeracy assistance is actively 
incorporated. 
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Appendix A: About the ALLS survey 
and the ACSF 



Adult Literacy and Life Skills survey 

The ALLS survey in Australia is a national survey that provides information on the knowledge and skills 
of 15 to 74-year-olds for: 

■ prose literacy: the ability to understand and use information from various kinds of narrative texts 

■ document literacy: the knowledge and skills reguired to locate and use information contained in 
various formats 

■ numeracy: the knowledge and skills reguired to effectively manage and respond to the 
mathematical demands of diverse situations 

■ problem-solving: goal- directed thinking and action in situations for which no routine solution is 
available. 

The ALLS survey administered tests on these and also sought the participants' self-assessment on prose 
and document literacy and numeracy. For each literacy domain, proficiency is measured on a scale 
ranging from 0 to 500 points, with these continuous scores grouped into five skill levels (only four levels 
were defined for the problem-solving scale) with level 1 being the lowest measured level of literacy. 
The survey collected a range of data on income and employment and also on social capital and 
wellbeing (ABS 2008, p.4). 

Australian Core Skills Framework 9 

The ACSF identifies five core skills, namely learning, reading, writing, oral communication and 
numeracy. Each skill is enacted for a purpose and the set of purposes, circumscribed as 'aspects of 
communication', includes personal, cooperative, procedural, technical, systems, and public 
communication. 

Performance in each core skill is described across five performance levels from level 1 (low) to level 5 
(high). At each performance level, indicators are used as broad descriptors. Performance is further 
elaborated through the use of 'performance features', which are grouped in strands called 'focus 
areas'. As an example, for reading the focus areas include purpose and audience, complexity, 
prediction and prior knowledge, text structure, textural analysis, critical literacy, textual clues, 
grammar and vocabulary. 

Four variables impact on performance at any point in time — the nature and degree of support, 
familiarity with the context, complexity of the text and complexity of the task. While it is expected 
that an individual may need high support during the initial learning phase, support should not be 
reguired in summative assessment. In the feasibility phase of the current study, experts were asked to 



9 Please note that at the time of writing the Australian Core Skills Framework was being reviewed. 
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assume that individuals would complete the tasks without support, thus, rendering the ALLS survey and 
the ACSF more directly comparable. 

A comparison of the performance levels and associated descriptors or performance features of the two 
frameworks is presented in table Al. 
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Table A1 Comparison of ALLS prose, document, numeracy and quantitative literacy (I ALS) and ACSF reading and numeracy performance level descriptors 



Performance 

level 



1 



2 



ALLS IALS 

Prose Document Numeracy Quantitative literacy Reading 



Most of the tasks in this 
level require the 
respondent to read a 
relatively short text to 
locate a single piece of 
information which is 
identical to or 
synonymous with the 
information given in the 
question or directive. If 
plausible but incorrect 
information is present in 
the text, it tends not to 
be located near the 
correct information. 



Some tasks in this level 
require respondents to 
locate a single piece of 
information in the text; 
however, several 
distractors or plausible 
but incorrect pieces of 
information may be 
present, or low-level 
inferences may be 
required. Other tasks 
require the respondent 
to integrate two or more 
pieces of information or 
to compare and contrast 
easily identifiable 
information based on a 
criterion provided in the 
question or directive. 



Tasks in this level tend 
to require the 
respondent either to 
locate a piece of 
information based on a 
literal match or to enter 
information from 
personal knowledge 
onto a document. Little, 
if any, distracting 
information is present. 



Tasks in this level are 
more varied than those 
in level 1 . Some require 
the respondents to 
match a single piece of 
information; however, 
several distractors may 
be present, or the match 
may require low-level 
inferences. Tasks in this 
level may also ask the 
respondent to cycle 
through information in a 
document or to integrate 
information from various 
parts of a document. 



Tasks in this level 
require the respondent 
to show an 

understanding of basic 
numerical ideas by 
completing simple tasks 
in concrete, familiar 
contexts where the 
mathematical content is 
explicit, with little text. 
Tasks consist of simple, 
one-step operations 
such as counting, sorting 
dates, performing simple 
arithmetic operations or 
understanding common 
and simple percentages 
such as 50%. 

Tasks in this level are 
fairly simple and relate 
to identifying and 
understanding basic 
mathematical concepts 
embedded in a range of 
familiar contexts where 
the mathematical 
content is quite explicit 
and visual with few 
distractors. Tasks tend 
to include one-step or 
two-step processes and 
estimations involving 
whole numbers, 
benchmark percentages 
and fractions, 
interpreting simple 
graphical or spatial 
representations, and 
performing simple 
measurements. 



Tasks in this level 
require the respondent 
to perform a single, 
relatively simple 
operation (usually 
addition) for which either 
the numbers are already 
entered onto the given 
document and the 
operation is stipulated, 
or the numbers are 
provided and the 
operation does not 
require the respondent 
to borrow. 



Tasks in this level 
typically require the 
respondent to perform a 
single arithmetic 
operation (generally 
addition or subtraction) 
using numbers that are 
easily located in the text 
or document. The 
operation to be 
performed may be easily 
inferred from the working 
of the question or the 
format of the material 
(e.g. a bank deposit form 
or an order form). 



Someone at level 1 
can read short simple 
texts of personal 
interest with an explicit 
purpose and 
containing highly 
familiar vocabulary. 

Appropriate reading 
tasks will involve one 
or two processes e.g. 
locating, recognising 
literal information. 



Someone at level 2 
can identify the main 
points in short, 
unambiguous written 
texts on familiar topics 
and using familiar 
vocabulary. 

Appropriate reading 
tasks will involve a 
limited number of 
familiar processes e.g. 
identifying, comparing 
and contrasting. 



ACSF 

Numeracy 

A person at this level can 
find and recognise key 
mathematical information in 
simple activities or texts, and 
apply simple mathematical 
and personal problem- 
solving strategies in very 
familiar contexts. They use 
everyday, informal language 
to express mathematical 
concepts. 

Appropriate numeracy tasks 
will involve one or two 
processes e.g. locating, 
recognising numerical 
information. 



A person at level 2 can 
identify and understand the 
relevant mathematical 
information in familiar texts 
or activities; can apply the 
appropriate problem-solving 
strategy in familiar contexts. 

Appropriate numeracy tasks 
will involve a limited number 
of familiar processes e.g. 
identifying, comparing and 
contrasting, such as 
comparing information in 
two-column tables. 
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Performance 

level 



ALLS 



Prose Document Numeracy 



3 



Tasks in this level tend 
to require respondents 
to make literal or 
synonymous matches 
between the text and 
information given in the 
task, or to make 
matches that require 
low-level inferences. 
Other tasks ask 
respondents to integrate 
information from dense 
or lengthy text that 
contains no 

organisational aids such 
as headings. 
Respondents may also 
be asked to generate a 
response based on 
information that can be 
easily identified in the 
text. Distracting 
information is present, 
but is not located near 
the correct information. 



Some tasks in this level 
require the respondent 
to integrate multiple 
pieces of information 
from one or more 
documents. Others ask 
respondents to cycle 
through rather complex 
tables or graphs which 
contain information that 
is irrelevant or 
inappropriate to the task. 



Tasks in this level 
require the respondent 
to demonstrate 
understanding of 
mathematical 
information represented 
in a range of different 
forms, such as in 
numbers, symbols, 
maps, graphs, texts, and 
drawings. Skills required 
involve number and 
spatial sense, 
knowledge of 
mathematical patterns 
and relationships and 
the ability to interpret 
proportions, data and 
statistics embedded in 
relatively simple texts 
where there may be 
distractors. Tasks 
commonly involve 
undertaking a number of 
processes to solve 
problems. 



These tasks require Tasks in this level Tasks at this level 

respondents to perform require respondents to require respondents to 

multiple-feature matches perform multiple-feature understand a broad 

and to integrate or matches, cycle through range of mathematical 

synthesise information documents, and information of a more 

from complex or lengthy integrate information; abstract nature 

passages. More however, they require a represented in diverse 

complex inferences are greater degree of ways, including in texts 

needed to perform inference. Many of these of increasing complexity 

successfully. Conditional tasks require or in unfamiliar contexts, 

information is frequently respondents to provide 

present in tasks at this numerous responses but 
level and must be taken don’t designate how 
into consideration by the many responses are 

respondent. needed. Conditional 

information is also 
present and must be 
taken into account by 
the respondent. 
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IALS 



ACSF 



Quantitative literacy Reading Numeracy 



Tasks at this level 
generally require a 
single operation but the 
operations are more 
varied i.e. some 
multiplication and 
division tasks. 
Sometimes two or more 
numbers are needed to 
solve the problem and 
the numbers are 
frequently embedded in 
more complex displays. 
Terms such as 'how 
many' and 'calculate the 
difference’ are still used, 
as in the lower-level 
tasks but some tasks 
require the respondent 
to make higher-order 
inferences to determine 
the appropriate 
operation. 



At level 3 an individual 
understands a range 
of texts on familiar 
subjects that may 
contain simple 
diagrams and charts. 
Identifies the main 
messages in longer 
routine texts requiring 
integration of a 
number of ideas and 
pieces of information 
and containing some 
specialised 
vocabulary. 

Appropriate tasks 
include a number of 
steps within one task 
e.g. sequencing, basic 
inferencing, 
extrapolation and 
integration. 



Someone at level 3 can find 
and interpret mathematical 
information that may be 
partly embedded in both 
familiar and less familiar 
tasks and texts; use a 
variety of mathematical and 
problem-solving strategies in 
both; use both informal and 
formal mathematical 
language. 

Appropriate numeracy tasks 
include a number of steps 
within one task, such as 
sequencing, basic inference, 
extrapolation and integration 
e.g. use technological 
devices appropriate to the 
work context to measure 
and record data and act on 
that information. 



Tasks at this level 
require the respondent 
to perform a single 
arithmetic operation 
where typically either the 
quantities or the 
operation are not easily 
determined i.e. for most 
tasks the question or 
directive does not 
provide a semantic 
relation term such as 
'how many’ or 'calculate 
the difference’ to help 
the respondent. 



Someone at level 4 
understands texts with 
relatively complex 
syntactic structures 
that may incorporate 
some technical 
specificity and 
information presented 
in graphic, 

diagrammatic or visual 
form. 

Texts may use 
specialised vocabulary 
and contain embedded 
information, 
abstraction and 



A person at level 4 can 
extract and evaluate 
mathematical information 
embedded in a range of 
tasks and texts; apply a 
range of mathematical and 
problem-solving strategies; 
appropriately use a range of 
informal and formal 
mathematical language. 



symbolism. 



Mapping adult literacy performance 



Performance 




ALLS 




IALS 




ACSF 


level 


Prose 


Document 


Numeracy 


Quantitative literacy 


Reading 


Numeracy 



4 cont ... 



5 



Some tasks in this level 
require the respondent 
to search for information 
in dense text which 
contains a number of 
plausible distractors. 
Others ask respondents 
to make high-level 
inferences or use 
specialised background 
knowledge. Some tasks 
ask respondents to 
contrast complex 
information. 



Tasks in this level 
require the respondent 
to search through 
complex displays that 
contain multiple 
distractors, to make 
high-level text-based 
inferences, and to use 
specialised knowledge. 



These tasks involve 
undertaking multiple 
steps to find solutions to 
problems and require 
more complex reasoning 
and interpretation skills, 
including 

comprehending and 
working with proportions 
and formulas or offering 
explanations for 
answers. 

Tasks in this level 
require respondents to 
understand complex 
representations and 
abstract and formal 
mathematical and 
statistical ideas, possibly 
embedded in complex 
texts. Respondents may 
have to integrate 
multiple types of 
mathematical 
information, draw 
inferences, or generate 
mathematical 
justification for answers. 



Tasks at this level 
require the respondent 
to perform multiple 
operations sequentially, 
and they must dis- 
embed the features of 
the problem from the 
material provided or rely 
on background 
knowledge to determine 
the quantities or 
operations needed. 



Appropriate reading 
tasks may involve 
complex task analysis 
involving application of 
a number of processes 
e.g. synthesising ideas 
and information within 
a text or from several 
texts, critical reflection 
and evaluation. 



Someone at level 5 
understands highly 
complex, lexically 
dense texts from a 
broad range of text 
types, including those 
incorporating a high 
level of technical 
specificity. 

Sophisticated task 
analysis e.g. selecting, 
synthesising and 
critically evaluating 
evidence, arguments 
and ideas from 
complex primary and 
secondary sources 
with highly embedded 
information. 



Appropriate numeracy tasks 
may involve complex task 
analysis involving 
application of a number of 
processes such as 
extracting, comparing and 
interpreting information e.g. 
a person works in a team to 
develop an operating budget 
for a project that involves 
multiple sources of income 
and expenses. 

A person at level 5 
understands, analyses and 
synthesises highly 
embedded information 
across a broad range of task 
or texts; chooses and 
applies highly complex 
mathematical and problem- 
solving strategies and 
communicates mathematical 
concepts using a wide range 
of informal and specialised 
language. 

Appropriate tasks are 
complex and include 
interpretation, analysis, 
reflection, synthesis, 
evaluation and 
recommendations e.g. 
applying statistical 
techniques to analyse data. 



Source ABS (f 996, pp. 1 1 3—1 9, 2008, p. 77); Department of Education, Employment and Workplace Relations (2008, pp. 42-58, If 2-33). 

Note: Only two dimensions of the ACSF performance criteria are shown; namely, the nature of the text and the complexity of the task. Level f of the ACSF includes a high level of external support, while 

performance at level 5 involves little or no support. By contrast, performance on ALLS items is entirely independent. 



NCVER 



20 



Appendix B: Delphi method and 
descriptions of ALLS items 

The Delphi method 

The so-called Delphi method represents a variety of approaches to seeking and reporting expert opinion 
on an issue. Variations of the method revolve around the extent to which, and at what stages of the 
process, experts share their opinions, with a view to reaching consensus j udgments (Linstone &Turoff 
2002 ). 

In the present study, the panel of experts was selected according to their knowledge of the ACSF, and in 
one case, for their knowledge of other literacy assessment practices. All panel members have extensive 
experience of literacy development and assessment, especially for adults. 

Panel members signed a confidentiality undertaking in relation to the ALLS and IALS items they would 
rate. 

Panel members were provided with information about the proj ect and specifically about the procedures 
that were to be followed in its execution. The documentation provided included information on the 
objectives of the first phase of the study and a brief outline of possible subseguent phases. Panel 
members were also provided with an initial set of five literacy items with instructions to rate these 
before attending a face-to-face meeting convened in mid-August 2010. 

At this meeting information about the proj ect that had been provided in advance of the meeting was 
reiterated. In particular, the relationship between the project and the evaluation of progress against 
Council of Australian Governments skills targets were emphasised. There was considerable discussion 
among panel members about the initial set of items, the ACSF levels and performance features. The 
discussion had the desired effect of bringing to the fore differences in interpretations of the 
performance descriptions of the ACSF, although there appeared to be considerable agreement in the 
performance levels assigned to items. 

Following the meeting, panel members were asked to rate an additional set of ten items independently 
and to return their ratings to the NCVER facilitators. After rating items, panel members were provided 
with information on the ALLS difficulty levels of the items that they assessed. 

Description of items 

The ALLS items used in this study are secure items; they cannot be released publicly. In order to provide 
readers with a sense of the items used, brief descriptions of them are provided. Not all of the items 
described below were provided to members of the expert panel. 
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Mapping adult literacy performance 



Item 



Description 



1 This item is presented as a letter advising patrons of a concert performance. It has some structured 
information about the performance and one paragraph of text. 

2 This item takes the form of an advertisement seeking support for a charity. It has a large-text title, a graphic 
and three short paragraphs, with the target information in the second paragraph. 

3 The prompt for this item is a one-page three-column brochure on a water treatment. The text is presented in 
relatively short paragraphs and the content is clearly signposted by the use of bold large font headings. 

4-6 Three forms that seek support for a charity are presented on a single sheet. The questions require 

information to be read from the three forms. The headings for the forms provide clues to direct the reader’s 
attention. 

7-8 The prompt for this item is an article that might occur in a newspaper or on a brochure on fire safety. It is in 

two columns on a single sheet. The text is separated using headings and bullet points. 

9 Information on oil consumption is presented graphically using two clearly labelled pie charts. The required 
information can be read directly from the labels of one of the charts. 

1 0 Information is presented as a newspaper article on nuclear waste in four columns. Instructions direct the 
test-taker's attention to a chart, in which information is presented with clearly labelled numbers. 

1 1 The prompt for this item is a job application form. Four pieces of information are provided and each 
corresponds to one item on the form. 

1 2 The prompt is the text of an article on growing a variety of decorative plant. It has a large bold-type heading 
and an introductory paragraph. It occupies about one-half of an A4 page in three columns. The text is 
divided into short paragraphs with bold headings. 

13-14 This is a well-spaced article on alcohol consumption. It includes several paragraphs of text, two lists and a 
table. The information required to answer the first question is presented directly in the first paragraph and 
the answer to the second question can be found at the end of the article. 

15-16 Information is presented in two graphs on fireworks. One, containing the information for the first question, is 
a simple line graph. The answer to the second question can be read from a stylised graph. 

1 7 Information is presented as a newspaper article in four columns in nine separate sections. Instructions 
direct the test-taker’s attention to a two brief articles identifiable by their headings. The task is to identify a 
sentence that contains target information. The second article and some of the information in the target 
article are distractors. 

1 8 Information about a job interview is presented as a pamphlet. The instructions for the task are simple and 
clear. Fleadings in the text help to direct the test-taker’s attention to the required information. 

N1 Information about election results for three candidates is presented. The information is brief and clearly set 

out. The task is to tally the total number of votes cast. 

N2 This is a short newspaper article about the amount of a particular toxin in breast milk. The prompt is a bar 

graph depicting the amount of toxin in breast milk in three time periods. There are two questions which 
require the respondent to describe changes in the levels of the toxin. Knowledge of percentages is required 
for one question. 
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